0.2.0: product hardening (versioning, config, typed errors, docs, packaging) - #4
Merged
Conversation
Introduce the product versioning contract, distinct from the trace schema version: - `trazo version` subcommand and `-version` flag report the product version (main.Version = 0.2.0), the supported trace schema version, and the git commit/build time read from the Go toolchain's build info (no ldflags). - docs/versioning.md documents the product-vs-schema version split, the schema compatibility contract (accept same-major, reject cross-major, what a 2.0.0 trace does), migration stance, and the release process. - CHANGELOG.md (Keep a Changelog) records this round and the 0.1.0 hardening. - Link the schema doc's Versioning section to docs/versioning.md. versionReport is kept pure and unit-tested for field rendering, commit truncation, the -dirty suffix, and the unknown-commit fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the bare {evaluations, fileErrors} JSON output with a versioned
envelope: outputVersion, trazoVersion, traceSchemaVersion, generatedAt, an
aggregate summary, results, and structured errors. The shape is a documented
contract (docs/output.md) with its own version, so CI and downstream tools can
depend on it and ignore unknown fields across minor bumps.
Introduce report.Summarize as the single source of the aggregate counts, shared
by the JSON summary and the text footer so they cannot disagree. GeneratedAt is
injected via report.Meta, keeping output deterministic in tests.
BREAKING (JSON output): top-level keys evaluations/fileErrors are now
results/errors under the envelope. Update the Python tests that parse the CLI
output and the golden file accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a stdlib-only `config` package and a `-config <path>` flag: a versioned JSON policy that pins which evaluators run and their thresholds, so a team can commit one file and get identical evaluation across dev and CI. No new dependency; the core stays stdlib-only. - Partial configs are valid (omitted fields keep defaults); unknown fields are rejected (a typo is an error, not a silent no-op); version is required. - Precedence: defaults < config file < explicitly-set flags (via flag.Visit), so the config is a reproducible baseline a one-off flag can still override. - The package is CLI-independent: config.Load + Config.Build return the evaluator set for use from another Go program. Document it in docs/config.md with an example policy, and record it in the changelog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….2.0 M4) Build the binary once in TestMain and exercise it end to end: clean exit 0, a bad finding exit 1, a file error exit 2, valid JSON envelope, Markdown FAIL report, missing path, invalid -format, two-PATH error, version subcommand, empty directory, and config/flag precedence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Classify why a file could not be evaluated with runner.ErrorKind (read_file, invalid_json, invalid_trace, evaluator, canceled) instead of leaving callers to match on message strings. The kind is set at each failure point in the runner, emitted as errors[].kind in the JSON output (outputVersion bumped to 1.1, an additive change), and tagged in the text and validate output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a -verbose flag that prints one metrics line to stderr after a run (loaded/valid/invalid/evaluated/duration), so batch and CI runs are observable without parsing the results. A trace-evaluation tool should itself be observable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a dedicated schema job that runs the Go schema-sync tests and validates the fixtures and emitter output against the published JSON Schema, so the schema is an active part of the contract rather than just documentation. Add a Go test that the shipped example config always loads, keeping the docs from drifting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add testdata/complex_run.json: one trace that interleaves two tool calls paired by id (no orphans), a tool error, a high-cost step, and a node visited enough times to look like a loop. An integration test runs the full structural evaluator set over it and asserts how the work divides (tool error -> bad, loop -> bad, cost overrun -> neutral, terminal end -> no node finding). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The M8 integration fixture testdata/complex_run.json is consumed by the Go integration test; add it to VALID_FIXTURES so the published JSON Schema is proven to accept it too, closing the cross-language loop for that fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docs/security.md: what a trace holds (opaque input/output/error payloads), the one component that sends data off-machine (the LLM judge, final output only), how to disable network egress, redaction at emit time, and not committing private traces. docs/llm-judge.md: the judge's opt-in flags and config keys, what it grades (last llm_call output, once), determinism (temperature 0, capped tokens), the timeout/cancellation model, no internal retries, and the deliberate split between a malformed verdict (neutral finding) and a transport failure (errors[].kind=evaluator, structural evaluators still run). Notes network-free testing via the fakeJudge Completion. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- LICENSE: MIT. - Makefile: build, check (vet+test), gate (local mirror of the CI evaluator gate), install, agent-test, and a cross-compiled `dist` target that produces per-platform archives (.tar.gz / .zip) plus a sha256 checksums.txt. - .github/workflows/release.yml: on a `v*` tag, guard that the tag matches the compiled-in const Version, vet+test, `make dist`, then publish the binaries to a GitHub release via `gh release create`. No third-party actions; stdlib-only ethos extended to CI. - README: status/release/go-version/license badges, an Install section (`go install`, from-source, prebuilt binaries with checksum verification), and a Core documentation section linking config/output/versioning/llm-judge/security. - .gitignore: ignore /bin and a bare ./trazo so ad-hoc builds do not dirty the vcs stamp; dist/ is already ignored. - CHANGELOG: record the integration fixture, the two new docs, and this packaging. Verified: cross-compiles clean for linux/darwin/windows (amd64/arm64) to static binaries; `trazo version` and the gate exit codes (0 clean, 1 bad) both correct. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second hardening round, taking the core from post-0.1.0 toward a releasable product. Branches off the merged 0.1.0. The trace schema version stays 0.1.0; this bumps the product version to 0.2.0.
What is in it (by milestone)
trazo versionsubcommand +-version; product-vs-schema versioning policy (docs/versioning.md).outputVersion,trazoVersion,traceSchemaVersion,generatedAt,summary,results,errors);docs/output.md. BREAKING (JSON): top-levelevaluations/fileErrors->results/errors.configpackage +-configpolicy file (stdlib-only JSON). Precedence: defaults < config < explicit flags (viaflag.Visit);docs/config.md+ example.runner.ErrorKind(read_file/invalid_json/invalid_trace/evaluator/canceled), surfaced aserrors[].kind(outputVersion 1.1, additive).-verboseoperational metrics to stderr.testdata/complex_run.json) exercised by the full evaluator set in Go and by the Python schema suite.docs/llm-judge.md(determinism, timeouts, no retries, malformed-verdict-vs-transport-failure) anddocs/security.md(what a trace holds, the one off-machine path, egress off by default, redaction, do not commit private traces).LICENSE(MIT),Makefile(build/check/gate/install/cross-compileddist+ checksums), tag-triggered release workflow (guards the tag against the compiled-inconst Version, publishes linux/darwin/windows amd64+arm64 binaries viagh, no third-party actions), README badges + Install + core-docs section.Precedence note
Flags override config (conventional 12-factor), pinned by
TestCLI_FlagOverridesConfig.Verification
go build ./... && go vet ./... && go test ./...— green (6 packages)pytest -qinagents/langgraph-reference— 36 passed (includes cross-language checks against the Go runner)trazo versionand the evaluator gate (exit 0 clean / 1 bad) verified.Core stays standard-library only (no
go.sum). SeeCHANGELOG.mdfor the itemized list.🤖 Generated with Claude Code